Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build nix itself somewhat incrementally, optionally #8543

Conversation

roberth
Copy link
Member

@roberth roberth commented Jun 19, 2023

Motivation

Add an extra build that produces a build of nix, built in nix, reusing components that will have been built by hydra.nixos.org.

This speeds up the build that's required when changes are made to just the manual, ie all within /doc.
Reviewing those has been frustrating because

  • in nix develop, you don't get incremental builds because doc prs tend to have different base commits.
  • nix build never matches anything in the binary cache

This leaves the normal build untouched.

The way the expression works is by building sets of files in separate derivations and then copying them in. The speedup is achieved when the intermediate derivations are cached.

Numbers

  • build a change in /doc: 0:42
  • rebuild everything: 8:24
  • a normal build including tests: TBD
  • 1 top level build and 8 component derivations which are mostly sequential

TODO:

  • normal build currently fails with src/libstore/tests/libnixstore-tests: error while loading shared libraries: libnixstore-tests.so: cannot open shared object file: No such file or directory.
    • I promised not to touch the regular build, and that's within reach (modulo one or two parameters)
  • what's up with rpath/rpath-link?
  • speed up the copying; bash impl is slow

Context

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

  • agreed on idea
  • agreed on implementation strategy
  • tests, as appropriate
    • functional tests - tests/**.sh
    • unit tests - src/*/tests
    • integration tests - tests/nixos/*
  • documentation in the manual
  • documentation in the internal API docs
  • code and comments are self-explanatory
  • commit message explains why the change was made
  • new feature or incompatible change: updated release notes

Priorities

Add 👍 to pull requests you find important.

@roberth roberth force-pushed the build-nix-itself-somewhat-incrementally branch from 617fc09 to 6afc540 Compare June 19, 2023 13:35
roberth added 4 commits June 19, 2023 16:27
Instead, use `$ORIGIN` which make the linker look for sibling files.
This shouldn't affect the regular build, but makes it possible to
reuse intermediate build projects in the optional incrementalized
build that's going to be added in the next commit.
Quoting the Hacking chapter:

This provides an extra package, `.#quickBuild`, which builds Nix using
individual derivations for the libraries and such. This allows cached versions
of those intermediate build steps to be used, so that it builds a lot quicker.

This package is useful for reviewing changes to the manual, assuming that the
base commit is in the binary cache.
The problem is rpaths are set to the build directory, which is not ok for an output.
@roberth
Copy link
Member Author

roberth commented Sep 29, 2024

The meson-based build implements this, and does so in a more precise and elegant way.
Closing 🎉

@roberth roberth closed this Sep 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant